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EXECUTIVE SUMMARY 



Each year, the National School Lunch Program (NSLP) and School Breakfast Program 
(SBP) serve almost 4 billion free and reduced-price meals to children from low-income 
households. Recent concerns about the integrity of the process used to establish eligibility for 
these benefits prompted the U.S. Department of Agriculture (USDA) to sponsor the NSLP 
Application/Verification Pilot Projects and to contract for an evaluation of two types of pilot 
approaches: (1) Up-Lront Documentation (ULD), and (2) Graduated Verification (GV). The 
evaluation is being conducted by Mathematica Policy Research, Inc. (MPR). 

One of several reports on the evaluation, this report focuses on issues relating to applications 
for free or reduced-price meals. In particular, we present (1) an impact analysis of the effects of 
the pilot policies on the number of applications submitted, and (2) a descriptive analysis of the 
accuracy of eligibility status determinations made by the School Pood Authorities (SPAs). The 
analyses address these main research questions: 



• How did the ULD and GV pilot interventions affect the percentage of students who 
applied for free or reduced-price meal benefits? In households that were income- 
eligible for benefits but did not apply for them, what reasons did parents give for not 
applying? 

• Were the SPAs’ determinations of eligibility status consistent with those calculated 
using data that households supplied in interviews conducted for the evaluation? Are 
differences due to administrative errors (disagreement between SPAs’ determinations 
and eligibility status calculated by MPR from the information provided on the 
application and associated documentation) or reporting differences (inconsistencies 
between information reported on surveys and that provided with the applications)? 
What are the specific reasons for these two types of discrepancies? 



STUDY DESIGN AND METHODOLOGY 

Districts using UPD required families to document their income or receipt of public 
assistance at the time they submitted their applications for free or reduced-price lunches. 
Districts then used this documentation to make eligibility determinations, but they did not verify 
any approved applications later in the school year. 

Districts using GV allowed families to use the standard application process, which does not 
require income documentation, but changed key aspects of the usual verification process. After 
verifying a small sample of approved applications, these districts verified additional applications 
if 25 percent or more of the applications in the initial test resulted in benefit reduction or 
termination. 
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The study used a comparison design, selecting households in districts participating in the 
pilot programs and in nonparticipating districts with similar characteristics. The data used in this 
report were collected from two sources: (1) household surveys conducted for the evaluation 
between October 2002 and January 2003, and (2) abstractions of information households 
submitted to SFAs between July and October 2002 with their applications for free or reduced- 
price meals. 

To measure the impacts of UFD and GV on the various application-related outcomes, we 
calculated the difference between the mean value of the outcome measure in pilot districts and 
the mean value in comparison districts, using regression methods to control for other differences 
between students in the districts. To investigate the accuracy of eligibility determinations, we 
compared three measures of meal price status: (1) the SFA’s determination, (2) an MPR 

determination based on application data, and (3) an MPR determination based on survey data. 

Two caveats should be kept in mind when interpreting differences between the data sources. 
First, although the survey was designed to collect household size and income information in a 
more thorough and consistent way than was possible using application forms, data from the 
survey may not accurately reflect circumstances during the period for which information was 
sought. Such reporting errors may be intentional or unintentional. Second, since in most cases 
the survey data were collected a few months later than the application data, some differences 
between data from these two sources may reflect changes in household circumstances over time 
rather than reporting error. 



FINDINGS ON THE IMPACTS OF PILOT INTERVENTIONS ON APPLICATION 
RATES 

The UFD pilot intervention resulted in a decrease in the probability that income-eligible 
households would apply for free or reduced-price meals. The UFD pilot was estimated to lead 
to a 7.4 percentage point decline in the percentage of students income-eligible for free or 
reduced-price meals who applied for benefits, all else equal. This estimate was statistically 
significant at the 0.05 level. The GV pilot led to a decline of 10.3 percentage points in the 
application rate among the same group; however, this estimate was not statistically significant at 
the 0.05 level. These effects, in turn, translated into impacts of roughly the same size on the 
probability that income-eligible students would become certified for benefits, which suggests 
that the primary reason eligible students did not become certified for free or reduced-price meals 
was that they did not complete an application. 

Neither UFD nor GV significantly influenced the likelihood that students from 
households income-ineligible for free or reduced-price meals would apply for or become 
certified for benefits. Among students not income-eligible for free or reduced-price meals, UFD 
and GV each had only a 0.3 percentage point impact on the likelihood that students would apply 
for benefits, and these estimates were not statistically significant. 

By far the most common reason that income-eligible households cited for failing to apply 
for free or reduced-price meals was that they did not know they were eligible for benefits. In 
UFD comparison districts, for example, two-thirds of income-eligible nonapplicants reported this 
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as their reason for not applying. The second most common situation was that some income- 
eligible nonapplicants believed (and reported on the survey) that they had applied even though 
there was no evidence in the administrative data of their having done so. 



FINDINGS ON THE ACCURACY OF ELIGIBILITY DETERMINATIONS 

For about 10 percent of noncategorically eligible households, the SFA’s eligibility 
determination differed from the eligibility status MPR computed from the application 
information provided. The percentage of applications with administrative errors is smaller 
(about 8 percent), if categorically eligible households are included. 

Errors in SFA determinations of eligibility from household application information were 
considerably more common in UFD pilot sites than in other districts. This reflected the fact 
that the most common type of administrative error by far — inaccuracy in computing total 
household income based on the information available — was considerably more prevalent in UFD 
sites than in their comparison districts or even in GV sites. The higher incidence of apparent 
administrative error in the UFD pilot sites probably is due to the fact that at those sites the 
calculations often had to be done directly from income source documents, which were more 
difficult to interpret than application forms. 

Reporting differences between application and survey data were considerably more 
widespread than administrative error. For about 36 percent of households overall, the 
eligibility status computed from the application information differed from that computed from 
the survey data. Information reported on applications matched that reported on surveys 
somewhat more often in pilot sites (although the differences are not statistically significant). 

The main source of overall inconsistencies appears to be household reporting rather than 
SFA administrative mistakes. When overall inconsistencies are assessed by comparing the SFA 
determination with eligibility as computed from survey data, about 90 percent of the differences 
between SFA certification decisions and survey-based eligibility determinations were a result of 
inconsistencies between information reported on the survey and information reported on the 
application. Only about 9 percent of differences resulted from administrative error in 
determining eligibility based on information submitted on the application. 

UFD did not materially improve the accuracy of income-reporting. About 34 percent of 
applications in UFD districts contained a discrepancy due to income-reporting, compared to 36 
percent of applications in UFD comparison districts. However, while UFD appears to have 
improved somewhat the accuracy of reporting on income sources that were acknowledged on the 
application form, it did not change the likelihood that a household would acknowledge either that 
a given person had income or that someone for whom income was reported had income from 
another source as well. Furthermore, discrepancies stemming from failure to report income of a 
person or source — the type of income-related difference least likely to be affected by the pilots — 
accounted for the largest share of income-related differences. 




I. INTRODUCTION 



The National School Lunch Program (NSLP) and School Breakfast Program (SBP) serve 
nearly 4 billion free and reduced-price meals annually to children certified as being from low- 
income households (U.S. Department of Agriculture 2004). In recent years, however, 
policymakers have raised concerns about the integrity of the process the programs use to 
establish eligibility for these benefits. In response, to test ways of improving the process for 
certifying students for free and reduced-price meals, the U.S. Department of Agriculture (USD A) 
asked school districts around the country to participate voluntarily in the National School Lunch 
Program Application/ Verification Pilot Projects. USDA also commissioned an evaluation of two 
of the approaches tested: (1) Up-Front Documentation (UFD), and (2) Graduated Verification 
(GV). 

This report presents (1) an impact analysis of the effects of the pilot policies on the number 
of applications for free and reduced-price meals submitted at the pilot sites, and (2) a descriptive 
analysis of the accuracy of meal price status determinations made by the School Food 
Authorities (SFAs) at the pilot and comparison sites. The impact analysis parallels the analysis 
of pilot program effects on other variables reported earlier (see below). The descriptive analysis 
is based on comparison of the SFA meal price status determinations with data made available in 
the application files and with data collected by Mathematica Policy Research, Inc. (MPR) 
through a survey of pilot and comparison site households. 

We present the findings of the evaluation project in five reports. In addition to this report, 
four companion reports present findings on: 

I . The impacts of UFD and GV on the certification of eligible and ineligible students for 
free or reduced-price benefits (Burghardt et al. 2004a) 
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2. A description of the study methods and supplementary tabulations (Burghardt et al. 
2004b) 

3. The impacts of the demonstration on NSLP participation — whether certified (and 
noncertified) students actually received school lunches (Gleason et al. 2004) 

4. An analysis of the operational aspects of the pilot projects, including the procedures 
used to implement the pilot policies and the costs associated with these procedures 
(Burghardt et al. 2004c)' 

The rest of this chapter presents important background information for the study. It also 
discusses the key research questions addressed. 

A. STUDY BACKGROUND 

Several studies examining income levels of students certified for free or reduced-price meals 
have found that a nontrivial number of these students’ families have income levels that make 
them ineligible for the level of benefits they are receiving (U.S. Department of Agriculture 1990 
and 1997). To address this issue, several school districts began testing alternative ways of 
determining the income eligibility of students’ families; the evaluation USDA commissioned 
focused on a subset of these districts. In particular, the evaluation included nine districts that 
tested UFD during the 2000-2001 through 2002-2003 school years and three districts that tested 
GV during these same years. ^ 



* In addition, under the same contract with the Food and Nutrition Service (FNS), MPR 
conducted a case study of verification outcomes in 21 larger metropolitan school districts that 
were not involved in the demonstrations in either the pilot or the comparison districts (Burghardt 
et al. 2004d). 

^ Two of the UFD districts operated the pilot in school years 2001-2002 and 2002-2003, and 
one of the GV districts operated the pilot in school years 2000-2001 and 2001-2002. 
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1. Demonstration Policies 



Under UFD, districts required that all applicants for free or reduced-price meals provide 
documentation of their income or evidence they were receiving benefits under the Food Stamp 
Program (FSP), Temporary Assistance for Needy Families (TANF), or the Food Distribution 
Program on Indian Reservations (FDPIR). If the application did not include documentation, it 
was considered incomplete, and a student could not receive benefits. After the district reviewed 
and made eligibility determination decisions about all complete applications, it was not required 
to perform the verification of income for the small sample of households called for in federal 
regulations. Students approved through direct certification in the UFD pilot districts were not 
subject to these requirements, which applied only to households that submitted an application. 

Under GV, application procedures were strengthened and, in certain circumstances, the 
verification process was enhanced. First, households that applied for free or reduced-price 
meals, and whose benefits had been terminated or reduced in the prior year because of the 
district’s verification procedures, were required to provide documentation of their incomes or of 
their categorical eligibility when they applied. Second, the district had to conduct the standard 
verification of 3 percent of approved households and the following additional verifications: 

• If 25 percent or more of the originally verified applications (among the 3 percent 
sample) led to a termination or reduction of free or reduced-price meal benefits, the 
district was required to verify an additional 50 percent of remaining applications. 

• If 25 percent or more of these second-round verifications resulted in terminations or 
reductions in benefits, the district was required to verify all remaining applications. 
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For additional details on the pilot projects and how their rules differed from standard 
district eligibility determination procedures, see Burghardt et al. (2004a). 
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B. RESEARCH QUESTIONS 

This report focuses on aspects of the free/reduced-price meal applications that the pilot and 
comparison sites received and examines how the relevant SFAs processed them. 

1. Impacts on Number of Applications 

In Chapter II, we draw on the pilot/comparison site design to assess the impacts of the UFD 
and GV pilot projects on the number of applications that pilot districts received. Key questions 
are: 

• What are the impacts of the pilot policies on the number of applications received at 
the pilot districts? 

• How does this vary for households whose incomes make them eligible for free or 
reduced-price meals and for households whose incomes make them ineligible for 
such meals? 

2. Descriptive Analysis of Application Accuracy 

The data assembled for the evaluation not only support the impact analysis, they also provide 
a rich basis for examining key aspects of how SFAs process applications. Key issues include: 

• How accurate are the SFA meal price status determinations? 

• When errors are made, what kinds of mistakes are most common? 

• Are certain types of cases more likely to have errors? 

• Are there differences in the findings for pilot versus comparison sites? 

We address each of these questions in this report, through descriptive analysis of the 
available data. As we examine these questions, the availability of multiple data sources both 
enriches and complicates our analysis. Figure I.l illustrates this. Three indications of free and 
reduced-price meal status are available to us. First, there is the actual determination made by the 
SFA, illustrated by apex 1 of the triangle in the figure. 
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FIGURE LI 



MEAL PRICE STATUS EROM THREE SOURCES 




Calculation by MPR based on application 
(free, reduced-price, paid) 



Calculation by MPR based on study interview 
(free, reduced-price, paid) 
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Second, as part of the data collection, we obtained and coded photocopies of the actual 
applications submitted by a sample of families at the study districts. Thus, we can calculate meal 
price status ourselves, using the same information available to the SFAs when they made their 
eligibility determinations. With the information submitted by the households, we can assess 
whether the SFAs processed the application information correctly. Essentially, this involves 
comparing apex 1 and apex 2 of the figure. In addition to the meal price category 
determinations, we compare the underlying data used by the SFAs and MPR to make the 
determinations. For example, is the monthly household income the SFA computed from the 
application materials equal to, higher than, or lower than the monthly household income MPR 
computed from the same materials or from the study interview? 

Finally, our data collection protocols also involved conducting detailed interviews with 
samples of households about key factors (mainly, income and household size) that determine 
their eligibility for free or reduced-price meals. These interviews supplied a third set of relevant 
data and allowed us to compare both the SFA decision (apex 1) and the information the 
households submitted to the SFA (apex 2) to the detailed interview data (apex 3). 

Each type of comparison provides a different and potentially useful perspective on issues 
related to meal price accuracy, and each is included in the analysis reported below. 

C. OVERVIEW OF REPORT 

Chapter II presents our analysis of the impacts of the pilots on submission of applications. 
We describe the pilot/comparison site methodology and report and discuss findings. 

The rest of the report discusses aspects of accuracy in determining meal price status, as 
discussed above. Chapter III examines the consistency between the actual SFA meal price 
determinations and the information the households submitted in their applications — essentially, 
the comparison between apex 1 and apex 2 in Figure 1. 1 . Chapter IV presents comparisons of 



6 




the application material the household submitted with the survey data (the comparison between 
apex 2 and apex 3). Chapter V examines the relationship between the interview data and the 
actual SFA meal price determinations (apex 3 versus apex 1). 



7 




II. IMPACTS OF THE PILOT INTERVENTIONS ON APPLICATIONS SUBMITTED 



The analysis we report in this chapter complements Burghardt et al. (2004a), which 
described the findings of the evaluation on the impacts of the pilot interventions on the 
certification of students for free or reduced-price benefits. That report concluded that the 
impacts of UFD and GV on certification among students income-ineligible for benefits were not 
statistically significant. In other words, the evaluation found no evidence that the pilots 
prevented income-ineligible students from being approved for free or reduced-price meals. On 
the other hand, the pilots reduced certification among income-eligible students, which suggests 
that the pilots created barriers to certification among some students who did qualify for benefits 
based on their household’s circumstances. 

Burghardt et al. (2004a), however, did not examine the impacts of the pilots on whether 
students applied for benefits. In this chapter, we present estimates of the impacts of UFD and 
GV on this measure. Estimating impacts on application rates will help us learn more about how 
these interventions influenced (or did not influence) students’ certification status. Among 
income-eligible students, for example, we will be able to determine the extent to which the pilots 
discouraged students from applying for benefits versus the extent to which students applied but 
had their applications rejected for some reason. 

Specifically, this chapter addresses the following questions: 

• How did UFD and GV affect the percentage of students who applied for free or 
reduced-price meal benefits? Did these effects differ for students who were income- 
eligible for free or reduced-price meals versus those who were not? 

• How did UFD and GV affect the percentage of students who applied for free or 
reduced-price meal benefits but whose applications were not approved? Did these 
effects differ for income-eligible versus income-ineligible students? 
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• Among students who were income-eligible for benefits but did not apply, what 
reasons did their parents give for not applying for benefits? Did UFD and GV have 
any effect on the prevalence of these reasons for not applying? 



A. DATA AND METHODS 

The basic design used to estimate the impacts of UFD and GV on applying for free or 
reduced-price meal benefits is the same as that used to estimate the impacts of the pilots on 
certification for these benefits, as described in Burghardt et al. (2004a). This is a comparison 
group design, in which a comparison district was selected for each of the UFD and GV pilot 
districts on the basis of having similar characteristics and being judged as a good match by local 
officials. We interviewed the parents of samples of students selected from each pilot and 
comparison district and used both survey and administrative data to measure their characteristics 
and key outcomes, such as whether they applied for benefits, became certified for benefits, and 
participated in the NSLP. The difference between mean outcomes in pilot versus comparison 
districts, after controlling for other characteristics in a regression framework, was the estimate of 
the impact of UFD and GV on these outcomes.^ 

The student samples on which the analysis was based were selected at random from among 
all students in the pilot and comparison districts, except for those who had been directly certified 
to receive free meals on the basis of receipt of food stamps, TANF, or assistance from the 
FDPIR. Thus, the results presented in this report can be generalized to all non-directly certified 

* A drawback of a nonexperimental comparison group design such as this is that it was 
difficult to identify and successfully recruit comparison districts that were similar to pilot 
districts in all ways except the pilot intervention. However, we did develop a careful process for 
selecting comparison districts that we felt would meet this objective, and the resulting 
characteristics of students in comparison versus pilot districts were, for the most part, similar. 
Nonetheless, we could not identify or explicitly control for relevant unobserved differences 
between the two groups of districts. See Burghardt et al. (2004a) for a description of the 
comparison district selection process, as well as the observed characteristics of students in pilot 
versus comparison districts. 
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students in the evaluation districts. The reason for the exclusion of directly certified students 
was that the UFD and GV pilot interventions directly affected only those students who could 
have applied for free or reduced-price meals. Students who were directly certified automatically 
received free meals without having to complete an application. Thus, the analysis of the impact 
of UFD and GV on applications excludes directly certified students. 

1. Measuring Student Applications 

The primary data source for measuring whether students applied for free or reduced-price 
meals was administrative data collected from the pilot and comparison districts in the evaluation. 
We submitted a list of the students in our sample and asked the districts to supply information on 
applications, if any, from those households in the period to be covered in the analysis. The 
districts provided MPR a copy of each application and any accompanying materials, including 
documentation of income for UFD pilot sites, for each student in our sample. Based on these 
data, we coded each household into one of the following categories: 

• Applied and was certified for free meals or for reduced-price meals 

• Applied and was denied 

• Applied and was neither certified nor denied (in which case, the application is 
assumed to have been incomplete) 

• Did not apply 

We are most confident in the distinction between households that applied versus those that did 
not apply and in the distinction between households certified versus those not certified. We have 



2 

In one district, MPR survey staff traveled to the site to examine the documents and abstract 
the information onto our coding form. 
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somewhat less confidence in our determination of whether the application was incomplete versus 
denied, since this determination could be due to missing data. 

In addition to the administrative data on applications, we collected survey data from parents 
on whether they applied for free or reduced-price meal benefits on behalf of their children. 
Because parents may not have recalled applying for these benefits, or they may have thought that 
they applied for free or reduced-price meals when they did not, we relied primarily on the 
administrative data on applications throughout most of the analysis. However, we did use the 
survey data on applications in two ways in this report. First, we tested the sensitivity of our 
estimates of the impact of UFD and GV on applications to the use of survey versus 
administrative data. Second, we used information provided on the survey to examine possible 
reasons that some income-eligible households failed to apply for free or reduced-price meal 
benefits. 

Ultimately, we used the administrative data to define the following outcome measures for 
the analysis: 



• Application Submitted — All Students: Percentage of all non-directly certified 

students who submitted an application for free or reduced-price meals"^ 

• Certified for Free or Reduced-Price Meals — All Students: Percentage of all non- 
directly certified students who both submitted an application and were certified to 
receive free or reduced-price meals 



-5 

Among the parents in UFD and GV districts who did apply for benefits according to the 
administrative data, about 3 percent claimed that they had not applied and, in fact, provided a 
reason for not applying. Among those who did not apply for benefits according to the 
administrative data, about 15 to 20 percent reported in the survey that they had applied. 

^ In this report, we sometimes refer to applications as having been submitted by students and 
sometimes as having been submitted by households. These terms are used interchangeably and 
are not intended to indicate the mode of application submission; we did not collect data on 
whether the form was delivered to the district by the student or by some other household 
member. 
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• Application Submitted and Not Approved — All Students: Percentage of all non- 
directly certified students who submitted an application for free or reduced-price 
meals but were not certified for benefits 

• Application Denied — All Students: Percentage of all non-directly certified students 
who submitted an application for free or reduced-price meals but whose application 
was denied 

• Application Incomplete — All Students: Percentage of all non-directly certified 

students who submitted an initial application for free or reduced-price meals but 
whose application presumably was incomplete because it was neither approved nor 
denied 



Each of the five outcome measures listed above was examined separately among students 
who are income-eligible for free or reduced-price meals — that is, with incomes of no more than 
185 percent of the federal poverty level — and among students ineligible for free or reduced-price 
meals — that is, with incomes of more than 185 percent of the federal poverty level. In addition, 
responses to the study interview provided data on reasons for not applying for free or reduced- 
price meals among students who are income-eligible but not certified for benefits. 

2. Approaches to Measuring Impacts 

To measure the impacts of UFD and GV on the application-related outcomes, we calculated 
the difference between the mean value of the outcome measure in pilot districts and the mean 
value of the measure in comparison districts, controlling for other differences between the two 
types of districts. The general model we estimated, which was similar to the one described in 
Burghardt et al. (2004a), is shown here: 

(1) y,=c + Xfi + f^d,DP,. *P,] + e , , 

j=2 k=l 

where: y, = application-related outcome for student i 

Xi = vector of characteristics of student i hypothesized to affect outcome y, 

DPij = binary indicator of whether student i attended the jth pilot-comparison 
district pair 
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Pi = binary indicator of whether student i attended a pilot district 
Ci = random-error term 

In this model, the outcome of interest (for example, measuring whether the student applied 
for free or reduced-price meals) is regressed on a set of student- level characteristics; a set of 
binary variables (called district pair variables), each of which represents a pilot district along 
with its matched comparison district; and a set of variables formed by interacting the district pair 
variables with a binary variable indicating whether the student’s district was a pilot district. The 
coefficients to be estimated in the model include a constant term c, a vector b, d 2 through dK, and 
ai through uk (where K represents the number of pilot districts that implemented UFD or GV), 
while Ci is a random-error term representing unobserved factors that influenced the outcome of 
interest. 

We estimated separate models to determine the impacts of UFD and GV. Since there are 
nine UFD pilot districts (K = 9), that model included eight binary district pair variables and nine 
district pair-pilot status interactions. There are three GV pilot districts (K = 3), so that model 
included two district pair variables and three district pair-pilot status interactions. 

The general model shown above allowed for differential effects of each of the pilot 
interventions in each of the districts in which it was implemented. For UFD, for example, the 
model produced nine different estimates of the impact of the pilot on a given outcome (the 
coefficients on the district pair-pilot status interactions, or ai through qk), each representing the 
estimated impact of the pilot in one of the districts in which it was implemented. To estimate the 
overall impact of UFD (or GV), we calculated the simple average of all the pilot district impact 
estimates. This manner of estimation gave equal weight to the effect of the pilot intervention in 
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each site, regardless of the size of the district or the number of students included in the sample 
from the district.^ 

3. Limitations of the Design 

Like all studies, this one is subject to several limitations that should be clearly understood. 
We note them here. 

a. Comparison Group Design 

We measured the impacts of the pilot projects on application-related outcomes by 
comparing the outcomes of families in the pilot districts with the outcomes of families in 
comparison districts. We took great care in identifying suitably matched comparison districts, 
and we controlled statistically for a range of personal and family characteristics that could 
influence the outcomes. As shown in Burghardt et al. (2004a), the comparison district matching 
process produced a set of comparison districts and a sample of families with characteristics 
similar to those of the pilot districts. However, while (in our judgment) these comparison 
districts provide a reasonable basis for measuring net impacts of the pilot interventions, an 
element of uncertainty, which we cannot quantify, remains about the quality of the benchmark. 

b. Sample Size Limitation 

As in most studies, constraints on resources limited the size of the samples it was possible to 
interview. For most outcomes, the study samples were large enough to give us confidence that if 
the pilot projects caused an impact of a policy-relevant magnitude — such as 20 to 30 percent of 

^ In calculating the standard error of the overall impact estimate, we took into account the 
fact that not all sample observations contributed equally to the overall estimate. In calculating 
the overall impact estimate, those observations from districts with larger-than-average samples 
were given a bit less weight than observations from districts with small samples. 
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the mean outcome in comparison districts — our sample would provide a high likelihood of 
detecting it. For some measures, however, limits on sample sizes constrain our ability to detect 
important demonstration effects. With only three GV pilot districts, compared with nine UFD 
pilot districts, the overall sample sizes for the GV models tended to be smaller than for the UFD 
models. Thus, we had less statistical power for detecting impacts of GV than we did of UFD. 

c. Issues of Generalizability 

The small number of demonstration sites and the voluntary nature of the decision to 
participate in the pilot projects necessarily limit our ability to draw conclusions about what 
would happen if the policies tested were to be implemented in a larger set of districts or 
nationwide. Only nine districts included in the study implemented UFD, and only three 
implemented GV. Furthermore, these districts were part of a very small group nationwide that 
volunteered to test new procedures designed to improve the accuracy of the process for 
administering NSLP certification. Burghardt et al. (2004a) document how these districts, as a 
group, compare with the nation, as a whole, in some readily observable characteristics. 
However, one can only speculate on how these districts differed from others nationwide in 
unobservable characteristics likely to affect the outcomes of interest in the evaluation. 

B. IMPACTS ON WHETHER HOUSEHOLDS APPLY FOR BENEFITS 
I. Impacts of UFD 
a. Preliminary Descriptive Analysis 

To examine the results of the analysis of how UFD influenced application rates for free or 
reduced-price meals, we first present, in Table II. 1, simple mean application and certification 
rates across the nine UFD pilot districts, as well as across the comparison districts. These results 
show the actual percentages of non-directly certified students in the two sets of districts that 
applied for and were approved for benefits, but we cannot infer an estimate of the impact of UFD 
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TABLE II. 1 



CERTIFICATION AND APPLICATION RATES IN UP-FRONT DOCUMENTATION PILOT AND 
COMPARISON DISTRICTS, BY HOUSEHOLD ELIGIBILITY STATUS 



Application/Certification Rate“ 
Comparison 

Student’s Application/Eligibility Status Pilot Districts Districts 



All Students 



Application submitted'’ 


20.7 


21.9 




(0.92) 


(0.65) 


Certified for free or reduced-price meals 


17.2 


20.3 




(0.80) 


(0.71) 


Application submitted and not approved 


3.5 


1.6 




(0.60) 


(0.39) 


Percentage of submitted applications that were approved^ 


83.1 


92.7 



Students Eligible for Free or Reduced-Price Meals (Income <185% FPL) 
Application submitted'’ 


48.6 


53.7 




(2.23) 


(1.73) 


Certified for free or reduced-price meals'' 


42.4 


50.5 




(2.15) 


(1.58) 


Application submitted and not approved 


6.1 


3.2 




(1.35) 


(1.10) 


Percentage of submitted applications that were approved"’ 


87.2 


94.0 



Students Not Eligible for Free or Reduced-Price Meals 






(Income >185% FPL) 






Application submitted'’ 


5.2 


5.1 




(0.65) 


(0.42) 


Certified for free or reduced-price meals"' 


3.8 


4.0 




(0.49) 


(0.47) 


Application submitted and not approved 


1.5 


1.1 




(0.44) 


(0.36) 


Percentage of submitted applications that were approved"’ 


73.1 


78.4 



Note; This table is intended to provide descriptive information on application/certification rates in UFD pilot and 
comparison districts only. Differences between application/certification rates in pilot and comparison 
districts should not be interpreted as estimates of the impact of UFD on NSLP these rates. See Table II.2 
for impact estimates. 

Excludes directly certified students. 

“Standard errors are in parentheses. These standard errors have been adjusted to account for the complex sampling 
design of the data set. 
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Table II. 1 (continued) 



’’The student was defined as having submitted an application if a copy of the application was found or if the student 
was approved for free or reduced-price meals even if no application was found. 

‘’Calculated by dividing the certification rate by the application rate. 

‘’As shown in Burghardt et al. (2004a), Tables IV. 1 and IV. 3. 

FPL = federal poverty level. 
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on the basis of these simple descriptive figures. Table II.2 presents impact estimates based on 
the regression model presented in Section II.A, along with regression-adjusted mean application 
and certification rates in pilot and comparison districts.^ 

In the preliminary descriptive analysis presented in Table II. 1, we focus on the comparison 
sites, since they reflect baseline conditions without the changes the pilot policies introduced. 
Among all non-directly certified students in UFD comparison districts, 21.9 percent submitted an 
application for free or reduced-price benefits in fall 2002 (Table II. 1). Most of these students — 
20.3 percent of all non-directly certified students — were approved by the SFA to receive free or 
reduced-price benefits. Only 1.6 percent of students submitted an application that was not 
approved, which amounts to about 7 percent of those who did submit an application.^ 



^ The regression-adjusted rates show the mean percentage of students in pilot and 
comparison districts that would have applied for and been approved for benefits if the 
characteristics of students across the two sets of districts had been equalized. 

•y 

Data from the household survey indicate that the percentage of parents who reported 
submitting an application was higher than the percentage who submitted an application 
according to administrative data. In UFD comparison districts, for example, 59.2 percent of 
parents of non-directly certified students eligible for free or reduced-price meals reported 
submitting an application (an additional 0.8 percent did not report submitting one but must have 
done so, because their children were certified for benefits), while the administrative data 
indicated that 53.7 percent actually submitted an application. Among ineligible students, 6.8 
percent either reported submitting an application or were certified, while the administrative data 
indicate that 5.1 percent actually submitted an application. Appendix Table A.l summarizes 
these results. 

^ We explored two possible reasons that submitted applications would not have been 
approved. First, the district may have denied the application after concluding that the household 
was not eligible for benefits on the basis of either income or receipt of food stamps, TANF, or 
FDPIR. Second, the application may not have been complete, and the household may not ever 
have provided all the required information or documentation. In UFD comparison districts, the 
data (not shown) suggested that most (about 76 percent) of the nonapproved applications were 
incomplete, as opposed to having been denied. In UFD pilot districts, a larger proportion of 
nonapproved applications were denied, but a majority were still incomplete. 
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TABLE II.2 



ESTIMATED IMPACT OE UP-ERONT DOCUMENTATION PILOT PROJECTS ON CERTIEICATION AND 
APPLICATION RATES AMONG STUDENTS NOT DIRECTLY CERTIEIED 



Student’s Application/Eligibility Status 


Regression-Adjusted Rate“ 
Pilot Comparison 


Impact’’ 


All Students 








Application submitted^ 


20.7 


22.0 


-1.3 

(1.35) 


Certified for free or reduced-price meals 


17.4 


20.1 


-2.7* 

(1.31) 


Application submitted and not approved 

Students Eligible for Eree or Reduced-Price 
Meals (Income <185% FPL) 


3.2 


1.9 


1.4 

(0.75) 


Application submitted*’ 


47.5 


54.9 


-7.4* 

(3.61) 


Certified for free or reduced-price meals** 


42.0 


51.1 


-9.1* 

(3.62) 


Application submitted and not approved 

Students Not Eligible for Eree or Reduced- 
Price Meals 
(Income>185% PPL) 


5.4 


3.7 


1.7 

(2.34) 


Application submitted*’ 


5.4 


5.1 


0.3 

(0.94) 


Certified for free or reduced-price meals** 


3.9 


3.9 


0.0 

(0.85) 


Application submitted and not approved 


1.5 


1.2 


0.3 

(0.56) 



Note: Excludes directly certified students. 

“The regression-adjusted rate shows the estimated application/certification rate that would result under the 
assumption that students in both UED pilot and comparison districts had the same distribution of observable 
characteristics. Thus, any remaining difference between the regression-adjusted rate in pilot-versus-comparison 
districts can be attributed to the impact of the pilot intervention. 

’’Standard errors are in parentheses. These standard errors have been adjusted to account for the complex sampling 
design of the data set. 

“The student was defined as having submitted an application if a copy of the application was found or if the student 
was approved for free or reduced-price meals even if no application was found. 

‘’As shown in Burghardt et al. (2004a), Tables IV.2 and IV.4. 

EPL = federal poverty level. 

*Significantly different from zero at the .05 level, two-tailed test. 
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Higher-income households were much less likely than lower-income households to apply for 
free or reduced price meal benefits. In particular, only 5.1 percent of students from households 
with incomes above 185 percent of the federal poverty level — and thus income-ineligible for free 
or reduced-price meals — applied for benefits, compared with more than half (53.7 percent) of 
those with incomes under 185 percent of poverty. Not surprisingly, a larger proportion of 
applicants who were income-eligible for benefits than those not eligible had their applications 
approved. Among the former group, 53.7 percent applied and 50.5 percent were approved — a 94 
percent approval rate. Among those not income-eligible for free or reduced-price meals, 5.1 
percent applied and 4.0 percent were approved — a 78 percent approval rate. 

Among households with students in UFD pilot districts, a smaller proportion applied for free 
or reduced-price meals than in comparison districts. Similarly, a smaller proportion were 
approved and certified for free or reduced-price meals. In contrast, the percentage of students in 
pilot districts who submitted an application that was not approved was larger than in comparison 
districts. The pilot-comparison district differences arise primarily from among students income- 
eligible for free or reduced-price meals. ^ As noted, however, we do not consider these 
descriptive pilot-comparison differences as estimates of the impact of UFD, since the differences 
could have been due to the pilot intervention, to differences in the characteristics of students in 
the two sets of districts, or to differences in the districts themselves. 

b. Impact Estimates 

Table II.2 provides estimates of the impacts of UFD on applications. Overall, while UFD 
was estimated to lead to lower application rates among all non-directly certified students, this 

^ See Appendix Table A.l for details on pilot-comparison differences in the percentage of 
parents who reported submitting an application. 
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impact was small and not statistically significant. Among students income-eligible for free or 
reduced-price meals, UFD had a larger negative impact on the application rate. All else equal, 
the pilot was estimated to lead to a 7.4 percentage point decline in the rate of students income- 
eligible for free or reduced-price meals that applied for benefits. This estimate was statistically 
significant at the 0.05 level. UFD had an even larger impact on the proportion of those income- 
eligible students who applied and were approved for (that is, were certified for) free or reduced- 
price meals. Table II.2 shows that the pilot led to a statistically significant decline of 9.1 
percentage points in the certification rate among this group, from 51.1 to 42.0 percent (with other 
factors held constant). Finally, the percentage of income-eligible students who submitted an 
application that was not approved was higher in pilot districts than in comparison districts, all 
else equal; but this difference was not statistically significant at the 0.05 level. 

Burghardt et al. (2004a) concluded that the UFD pilots created barriers to free or reduced- 
price meal certification among some income-eligible students. That report, however, did not 
explore the reasons for these barriers. Using the application data available here, we can 
distinguish between two possible explanations for the estimated impacts of UFD on certification 
among students income-eligible for benefits. In particular, the impacts on certification may have 
arisen because UFD discouraged income-eligible students from submitting an application for 
benefits. Alternatively, the pilots could have led to a larger proportion of income-eligible 
applicants applying but not being approved for benefits. This failure of having their benefits 
approved could have been due to the applicants not completing the application process by 
submitting all the required documentation or having their completed applications denied by the 
district. 

Evidence from Table II.2 suggests that the primary reason income-eligible students did not 
become certified for free or reduced-price meals was that their households did not complete 
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applications. The UFD pilot had a statistically significant 7.4 percentage point impact on the 
percentage of income-eligible students applying for benefits but only a 1.7 percentage point 
impact (that was not statistically significant) on the percentage that submitted applications but 
were not approved by the district. 

Among students not income-eligible for free or reduced-price meals, UFD had little effect 
on either the likelihood that they would apply for benefits or that their applications would be 
approved. Among all income-ineligible students, the estimated impact of UFD on the 
application rate was 0.3 percentage points and not statistically significant. The estimated impact 
on the likelihood that an income-ineligible student would apply for benefits but not have the 
application approved was also 0.3 percentage points and not statistically significant. 
Consequently, the pilot was estimated to have no impact on the certification rate among students 
income-ineligible for free or reduced-price meals. 

2. Impacts of GV 
a. Preliminary Descriptive Analysis 

In general, the GV districts had higher application rates than the UFD districts (34.2 percent, 
compared to 20.7 percent). Among non-directly certified students in GV comparison districts, 
39.0 percent submitted an application for free or reduced-price benefits in fall 2002 (Table II.3). 
Most of these students — 36.1 percent of all non-directly certified students, or 93 percent of those 



As discussed in Burghardt et al. (2004a), the GV districts had larger proportions of 
students from households with income levels below 185 percent of the FPL. 
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TABLE II.3 



CERTIFICATION AND APPLICATION RATES IN GRADUATED VERIFICATION PILOT AND 
COMPARISON DISTRICTS, BY HOUSEHOLD ELIGIBILITY STATUS 



Application/Certification Rate“ 
Comparison 

Student’s Application/Eligibility Status Pilot Districts Districts 



All Students 



Application submitted'’ 


34.2 


39.0 




(1.80) 


(1.79) 


Certified for free or reduced-price meals 


30.5 


36.1 




(1.53) 


(1.67) 


Application submitted and not approved 


3.1 


2.9 




(0.98) 


(0.84) 


Percentage of submitted applications that were approved"’ 


89.2 


92.6 



Students Eligible for Free or Reduced-Price Meals (Income <185% FPL) 
Application submitted'’ 


61.5 


75.4 




(3.36) 


(3.26) 


Certified for free or reduced-price meals"' 


60.0 


72.1 




(3.33) 


(2.98) 


Application submitted and not approved 


0.9 


3.3 




(0.51) 


(1.56) 


Percentage of submitted applications that were approved"" 


97.6 


95.6 



Students Not Eligible for Free or Reduced-Price Meals 






(Income >185% FPL) 






Application submitted'’ 


10.1 


13.0 




(2.02) 


(2.38) 


Certified for free or reduced-price meals"' 


4.8 


9.7 




(0.71) 


(2.13) 


Application submitted and not approved 


5.2 


3.3 




(1.92) 


(1.19) 


Percentage of submitted applications that were approved"" 


47.5 


74.6 



Note; This table is intended to provide descriptive information on application/certification rates in GV pilot and 
comparison districts only. Differences between application/certification rates in pilot and comparison 
districts should not be interpreted as estimates of the impact of GV on these rates. See Table II.4 for 
impact estimates. 

The percentage certified and the percentage submitted but not approved may not sum to the total 
percentage submitted, due to missing data. For several households known to have not been certified to 
receive free or reduced-price meals, data on whether the household submitted an application is missing. 

Excludes directly-certified students. 
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Table II. 3 (continued) 



“Standard errors are in parentheses. These standard errors have been adjusted to account for the complex sampling 
design of the data set. 

’’The student was defined as having submitted an application if a copy of the application was found or if the student 
was approved for free or reduced-price meals even if no application was found. 

“Calculated by dividing the certification rate by the application rate. 

‘’As shown in Burghardt et al. (2004a), Tables IV.9 and IV.ll. 

FPL = federal poverty level. 
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who submitted an application — were approved for benefits. As in UFD districts, the percentage 
of GV students who submitted an application that was not approved was small (2.9 percent). 

About three-fourths (75.4 percent) of non-directly certified students income-eligible for free 
or reduced-price meals submitted an application, and nearly all were approved for benefits (72.1 
of 75.4 percentage points, or 96 percent). The application rate among students not income- 
eligible for benefits was much lower, but nontrivial — about one in eight (13.0 percent) income- 
ineligible students submitted an application. Among these income-ineligible applicants, 73 
percent were approved for benefits (9.7 of 13.0 percentage points). 

Among students in GV pilot districts, a smaller proportion of those non-directly certified 
applied for free or reduced-price meals than in comparison districts. Similarly, a smaller 
proportion were approved and certified. These differences held among both students who were 
income-eligible and those who were not income-eligible for free or reduced-price meals. Among 
income-eligible students, for example, 61.5 percent of those in pilot districts submitted 
applications, compared with 75.4 percent of those in comparison districts (Table II.3). Among 
income-ineligible students, the application rates were 10.1 percent in pilot districts and 13.0 
percent in comparison districts. 

One difference between income-eligible and ineligible students in GV pilot districts was the 
large disparity in the approval rates of eligible versus ineligible students. Among income- 
eligible students who submitted an application, nearly all (98 percent) were approved. Among 



" In GV comparison districts, among applications that were not approved, we found 
evidence that 57 percent were denied. This suggests that households never completed the 
remaining 43 percent of nonapproved applications. In GV pilot districts, we found evidence that 
more than three-fourths (78 percent) of the nonapproved applications were denied, with the rest 
presumably incomplete. See Appendix Table A. 2 for additional details. 
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income-ineligible students, by contrast, fewer than half (48 percent) of those who applied for 
benefits were approved. 

b. Impact Estimates 

Once we controlled for student characteristics, GV was estimated to have led to a decline of 
10.3 percentage points in the application rate among students income-eligible for free or 
reduced-price meals (Table II.4). Although this estimate is not statistically significant at the 0.05 
level, it is relatively large. Along with fewer income-eligible students applying for benefits, GV 
also led to a decline (of 9.4 points) in the percentage approved for free or reduced-price meals 
(also not statistically significant at the 0.05 level). Finally, the percentage of income-eligible 
students who submitted an application that was not approved was lower in pilot districts than in 
comparison districts, all else equal (again this difference was not statistically significant at the 
0.05 level). 

As with UFD, Burghardt et al. (2004a) concluded that GV created barriers to free or 
reduced-price meal certification among some income-eligible students. The estimated impact, 
presented in Table II.4, of GV on application and certification rates among income-eligible 
students suggests that, essentially, the entire impact of GV on certification for free or reduced- 
price meals came about because GV discouraged income-eligible students from submitting an 
application for benefits. This conclusion is based on the fact that the negative effect of GV on 
the application rate (10.3 percentage points) is close in magnitude to the negative effect of the 
pilot on the certification rate (9.4 percentage points). 
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TABLE II.4 



ESTIMATED IMPACT OE GRADUATED VERIEICATION PILOT PROJECTS ON CERTIEICATION AND 
APPLICATION RATES AMONG STUDENTS NOT DIRECTLY CERTIEIED 



Regression-Adjusted Rate“ 

Student’s Application/Eligibility Status Pilot Comparison Impact’’ 



All Students 

Application submitted*’ 

Certified for free or reduced-price meals 

Application submitted and not approved 

Students Eligible for Eree or Reduced-Price 
Meals (Income <185% FPL) 

Application submitted*’ 

Certified for free or reduced-price meals** 

Application submitted and not approved 

Students Not Eligible for Eree or Reduced- 
Price Meals 
(Income>185% PPL) 

Application submitted*’ 

Certified for free or reduced-price meals** 

Application submitted and not approved 



34.6 


38.8 


- 4.2 


30.8 


35.9 


(3.02) 

-5.2 


3.4 


2.6 


(2.73) 

0.8 






(1.60) 


63.5 


73.8 


-10.3 


61.6 


71.0 


(5.31) 

-9.4 


1.3 


3.0 


(5.35) 

-1.7 






(1.48) 



11.8 


11.5 


0.3 

(2.93) 


6.1 


8.6 


-2.5 

(1.75) 


5.6 


2.9 


2.7 

(2.48) 



Note: Excludes directly-certified students. 

“The regression-adjusted rate shows the estimated application/certification rate that would result under the 
assumption that students in both GV pilot and comparison districts had the same distribution of observable 
characteristics. Thus, any remaining difference between the regression-adjusted rate in pilot versus comparison 
districts can be attributed to the impact of the pilot intervention. 

’’Standard errors are in parentheses. These standard errors have been adjusted to account for the complex sampling 
design of the data set. 

“^The student was defined as having submitted an application if a copy of the application was found or if the student 
was approved for free or reduced-price meals even if no application was found. 

**As shown in Burghardt et al. (2004a), Tables IV. 10 and IV. 12. 

EPL = federal poverty level. 

*Significantly different from zero at the .05 level, two-tailed test. 

**Significantly different from zero at the .01 level, two-tailed test. 
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C. REASONS SOME INCOME-ELIGIBLE HOUSEHOLDS DO NOT APPLY FOR 

BENEFITS 

In UFD and GV districts, a substantial percentage of students income-eligible for free or 
reduced-price meals did not apply for free or reduced-price benefits. Furthermore, the UFD and 
GV pilot interventions were both estimated to reduce the likelihood of applying. Since free or 
reduced-price meals potentially provide an important financial benefit to low-income 
households, these findings lead to two related questions. First, why do some income-eligible 
students not apply for benefits? Are they unaware of the existence of, or their eligibility for, 
these benefits? Is it that being certified for free or reduced-price meals is not actually an 
important benefit to them because they do not typically eat school meals? Or do the 
nonmonetary costs of applying for (or receiving) benefits exceed the monetary rewards? Second, 
what was it about the pilots that led larger proportions of income-eligible students not to apply 
for benefits? 

To answer the first question, we examined data collected as part of the household survey on 
nonapplicants’ reasons for not applying for benefits. We focused on students from households 
that were income-eligible for free or reduced-price meals but did not apply for certification and 
reported a reason for not applying.'^ We provided survey respondents with a variety of options 
as reasons for not applying and allowed them to select as many responses as were relevant. 



1 2 

A small proportion of eligible households reported a reason for not applying for benefits 
even though administrative data suggested that they had applied. A somewhat larger group (15 
to 20 percent) of eligible households reported that they applied for benefits even though the 
administrative data suggested that they had not applied. Among the circumstances that could 
have created this situation are the following: (1) the household submitted an incomplete 

application and the district never recorded it, (2) the district lost the application, (3) our staff 
failed to record the application that was present, or (4) the household respondent was mistaken 
about having applied for benefits. 

I 

Few respondents — less than 10 percent of eligible nonapplicants — gave more than one 
reason for not applying. 
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Tables II.5 and II.6 show the reported reasons for not applying among income-eligible 
nonapplicants in UFD and GV districts, grouped into categories of related reasons. 

As in the earlier descriptive analysis, we begin by examining the comparison site data. The most 
common reason income-eligible households reported for not applying for free or reduced- price 
meals was that they did not believe they were eligible for them. In UFD comparison districts, for 
example, two-thirds of income-eligible nonapplicants reported not applying for this reason 
(Table II.5). Only a handful of respondents — 1.3 percent in UFD comparison districts — were not 
aware of the availability of free or reduced-price meals. The second most common reason was 
that the student did not eat school meals, so it did not make sense for the household to apply 
(15.3 percent in UFD comparison districts). A few income-eligible nonapplicant households 
reported that they did not want to receive government assistance (7.8 percent) or that they 
wanted to avoid the stigma associated with free or reduced-price meals (7.5 percent). However, 
in UFD comparison districts, few households said they did not apply either because of the hassle 
of the application process (4.8 percent) or for concerns about privacy (0.6 percent). No 
households cited a desire to avoid the verification process as a reason for not applying for 
benefits. 

The distribution of reported reasons for not applying for benefits in GV comparison districts 
was similar to the distribution of reported reasons in UFD comparison districts. Again, two- 
thirds of income-eligible nonapplicants reported they did not know they were eligible for 
benefits (Table II.6). About 16.1 percent reported that their child did not eat school meals, and 
8.6 percent reported privacy concerns. 
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TABLE II.5 



REPORTED REASONS EOR NOT APPLYING EOR EREE/REDUCED-PRICE MEAL BENEEITS AMONG 
INCOME-ELIGIBLE NONAPPLICANTS IN UP-FRONT DOCUMENTATION PILOT AND 

COMPARISON DISTRICTS 





Percentage of Households Citing Reason 




Pilot 


Comparison 




Districts 


Districts 


Reasons for Not Applying Among Non-Directly Certified 
Households That Did Not Apply“ 






Costs of Applying for Benefits 






Wanted to avoid hassle of application process 


7.5 


4.8 




(3.61) 


(2.29) 


Did not want to reveal private information to school district 


0.0 


0.6 


officials 


(0.0) 


(0.71) 


Costs of Receiving Benefits 






Did not want to receive government assistance 


6.7 


7.8 




(3.08) 


(2.78) 


Wanted to avoid stigma associated with receiving free/reduced- 


7.3 


7.5 


price meals 


(3.16) 


(2.82) 


Benefit Not Worthwhile 






Child does not eat school meals 


14.3 


15.3 




(3.72) 


(3.36) 


Unaware of Eligibility/Benefits 






Did not think they were eligible 


65.4 


66.6 




(5.70) 


(4.94) 


Not aware of availability of free/reduced-price benefits 


5.6 


1.3 




(2.85) 


(0.86) 


Other Reasons for Not Applying 






Did not want to be selected for verification 


0.0 


0.0 




(0.0) 


(0.0) 


Other 


4.2 


7.0 




(1.94) 


(2.70) 


Sample Size 


97 


104 



Note; Tabulations are weighted. Estimates are average of district levels. 

“Column percentages may sum to greater than 100, because respondents could give more than one reason. 
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TABLE II.6 



REPORTED REASONS EOR NOT APPLYING EOR EREE/REDUCED-PRICE MEAL BENEEITS AMONG 
INCOME-ELIGIBLE NONAPPLICANTS IN GRADUATED VERIEICATION PILOT AND 

COMPARISON DISTRICTS 





Percentage of Households Citing Reason 




Pilot 


Comparison 




Districts 


Districts 


Reasons for Not Applying Among Non-Directly Certified 
Households That Did Not Apply“ 






Costs of Applying for Benefits 






Wanted to avoid hassle of application process 


2.6 


0.0 




(1.82) 


(0.0) 


Did not want to reveal private information to school district 


2.4 


8.6 


officials 


(3.04) 


(6.65) 


Costs of Receiving Benefits 






Did not want to receive government assistance 


8.2 


6.3 




(4.46) 


(4.69) 


Wanted to avoid stigma associated with receiving free/reduced- 


5.8 


7.2 


price meals 


(3.70) 


(5.08) 


Benefit Not Worthwhile 






Child does not eat school meals 


7.2 


16.1 




(3.82) 


(7.30) 


Unaware of Eligibility/Benefits 






Did not think they were eligible 


78.7 


67.2 




(6.51) 


(8.53) 


Not aware of availability of free/reduced-price benefits 


0.0 


3.5 




(0.0) 


(4.10) 


Other Reasons for Not Applying 






Did not want to be selected for verification 


0.0 


0.0 




(0.0) 


(0.0) 


Other 


3.7 


9.6 




(3.27) 


(5.85) 


Sample Size 


43 


23 



Note; Tabulations are weighted. Estimates are average of district levels. 

“Column percentages may sum to greater than 100, because respondents could give more than one reason. 
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Tables II.5 and II.6 also report reasons that income-eligible nonapplicants in UFD and GV 
pilot districts gave for not applying for free or reduced-price meals. The reported reasons in 
UFD pilot districts follow a pattern similar to those in UFD comparison districts. In GV pilot 
districts, income-eligible nonapplicants were a bit more likely than those in GV comparison 
districts to report that they did not think they were eligible for benefits. However, the pilot- 
comparison differences reported in these tables may be misleading as indicators of how UFD and 
GV affected the reasons for not applying for benefits, for two reasons. First, these differences 
have not been regression-adjusted for differences in the characteristics of students in the pilot 
and comparison districts. Second, and potentially more important, the samples on which Tables 
II.5 and II.6 are based were restricted to income-eligible students who did not apply for free or 
reduced-price meals. Since the earlier analysis in this chapter suggests that both pilots may have 
influenced households’ decisions to apply for benefits, restricting the sample in this way in 
looking at the reported reasons for not applying may have the result of hiding pilot-comparison 
differences that exist among the full group of income-eligible households. 

Thus, to estimate the impacts of UFD and GV on households’ reasons for not applying for 
benefits, we examined the sample of all households with students income-eligible for free or 
reduced-price meals, using regression techniques similar to those used for the other impact 
estimates. For this sample, we looked for differences between pilot and comparison districts in 
the percentage of households who had particular reasons for not applying for free or reduced- 
price meals and who did not apply as a result. To complete the picture, we also looked for 
differences in the percentage of households who applied for benefits, as well as the percentage 
who did not apply but reported that they did apply. In other words, all income-eligible students’ 
households were grouped into three categories: (1) applied for benefits, (2) did not apply for 
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benefits based on administrative data but reported applying in their responses to the survey, and 
(3) did not apply for benefits and reported a reason/"^ 

We further decomposed this third group into the five different types of reasons given for not 
applying (shown in Tables II.5 and II.6) and prioritized those reasons (as described below) so 
that each student would fall into only one category in terms of the reported reason. Because we 
were most interested in the reasons for not applying that were potentially related to the pilot 
intervention, we gave the highest priority to the reasons related to the costs of the application 
process. In other words, nonapplicants from households below 185 percent of the FPL who 
reported either that they “wanted to avoid the hassle of the application process” or “did not want 
to reveal private information to school district officials” were coded as having not applied 
because of the “costs of applying for benefits.” We ignored any additional reasons these 
respondents reported for not applying. If they did not report either of these reasons, however, we 
examined the reasons that were given the next highest priority; and, if they gave one of these 
second-highest-priority reasons, we ignored any additional reasons reported. Overall, we 
prioritized the reasons for not applying for free or reduced-price meals as follows: 



1. Costs of Applying for Benefits: Wanted to avoid hassle of application process OR 
did not want to reveal private information to school district officials 

2. Costs of Receiving Benefits: Did not want to receive government assistance OR 
wanted to avoid the stigma associated with receiving free or reduced-price meals 

3. Benefit Not Worthwhile: Child does not eat school meals 

4. Unaware of Eligibility/Benefits: Did not think they were eligible for benefits OR 
were not aware of availability of free or reduced-price meal benefits 

5 . Other Reasons for Not Applying 



Households were categorized as having applied for benefits if a copy of their application 
was found or if they were approved for free or reduced-price meals even if no application was 
found. 
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Once we had categorized all households with incomes below 185 percent of the FPL in our 
sample into seven mutually exclusive categories: (1) having applied for free or reduced-price 
meals, (2) not having applied but reporting an application, or (3 through 7) not having applied for 
one of the five reasons listed above, we then estimated the impacts of UFD and GV on these 
outcomes. In doing so, we ran a separate regression for each of the seven categories, controlling 
for relevant household characteristics using the general regression model described in Section A. 
The resulting estimates of the impacts of the pilots on income-eligible students’ application 
status/reasons for not applying are provided in Table II.7 for UFD and Table II.8 for GV. As 
shown in Table II.2, Table II.7 indicates that UFD had a significant negative impact on the 
percentage of income-eligible households who applied for free or reduced-price meals, lowering 
the application rate by 7.4 percentage points. However, the pilot did not significantly influence 
the prevalence of any of the potential reasons for not applying for benefits. All else equal, for 
example, the percentage of income-eligible households that did not apply because of the costs of 
applying for benefits or because they did not want to receive the benefits (that is, because of the 
costs of receiving the benefits) was low and not significantly different in pilot versus comparison 
districts. The percentage of households that did not apply because they were not aware of their 
eligibility for benefits (or of the benefits themselves) was a bit higher in UFD pilot districts than 
in comparison districts, but this difference was not statistically significant. Similarly, a larger 
percentage of income-eligible students in pilot districts than in comparison districts reported that 
they applied for benefits, even though the administrative data had not recorded an application. 
One possible explanation for this difference is that the pilot’s requirements that applicants submit 
documentation of the information reported on their application could have resulted in a larger 
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